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Are Public or Private Schools Doing Better? 
How the NCES Study is Being Misinterpreted 
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A recent study published by the National Center for 
Education Statistics (NCES) contains some surprising 
results based on a snapshot of student achievement 
data. According to the study, public school students are 
performing better than private school students in 
fourth grade mathematics and at the same level as pri- 
vate school students in fourth grade reading and eighth 
grade math. Indeed, the report says that private school 
students have an advantage over public school stu- 
dents only in eighth grade reading,^ These results 
should be handled very carefully. 

Policymakers and journalists need to know that the 
NCES findings that public schools outperform private 
schools employ significantly limited data. Some com- 
mentators on the NCES report appear to believe that 
this study describes causal relationships — that public 
school attendance causes better student achievement 
and that private school attendance causes students to 
have lower math and reading achievement. 

The NCES study analyzes the 2003 National 
Assessment of Educational Progress (NAEP) data that, 
due to a major limitation, are ill-suited for making 
any causal inferences. The NAEP data assess achieve- 
ment only at one point in time, providing a snapshot 
of how American students are performing in math 
and reading at that specific time. The NAEP data are 
not suitable for evaluating the effectiveness of private 
or public school attendance in raising academic 
achievement. In fact, the NCES authors explicitly 
warn against this in two sections of the report that are 
appropriately titled “Cautions in Interpretation.”^ 
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Talking Points 

• A recent study published by the National 
Center for Education Statistics argues that 
public school students perform better than 
private school students. These results should 
be Interpreted cautiously. 

• The National Assessment of Educational 
Progress data used in the report provide a 
snapshot of student achievement and can be 
used only to describe the characteristics of 
the student population at the time of the test. 

• The NCES report is being used to discredit 
private school voucher programs. However, 
studies based on better methods show that 
students who attend private schools 
through a voucher program experience 
greater achievement gains than do their 
public school counterparts. 

• Voucher programs are a powerful school 
choice tool, and more families should have 
the opportunity and flexibility to choose the 
most effective school for their children to 
achieve success. 
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Furthermore, various individuals and organiza- 
tions have used the report to discredit private 
school voucher programs,^ asserting that govern- 
ment education funds should be used only to fund 
public education, where they can be put to better 
use. However, the results of several more sophisti- 
cated and conclusive studies of voucher program 
effectiveness overwhelmingly point toward greater 
math and reading achievement for students who 
attend private schools through voucher programs. 

Causal Interpretation 

As noted, some commentators appear to believe 
that the NCES study describes causal relation- 
ships — that public school attendance causes better 
student achievement and private school attendance 
causes students to have lower math and reading 
achievement. This is surprising because the authors 
of the report explicitly warn against causal interpre- 
tations. They also acknowledge that comparisons of 
public and private schools (which amounts essen- 
tially to comparing apples to oranges) may not be 
useful because of the differences that exist between 
and within the public and private school populations. 

The NCES report analyzes the 2003 NAEP data, 
a nationally representative assessment of academic 
performance in math, reading, and science (among 
other things). Researchers frequently use the NAEP 
data to gauge student academic performance in the 
United States, even though the cross-sectional 
nature of these data limits what these researchers 
can do. Analysis of NAEP data can only be used to 



generally describe the characteristics of students 
who took the NAEP tests. These data are not suit- 
able for use in evaluating the effectiveness of public 
and private schooling over time. 

In fact, the NCES study would not even meet the 
standards of effective research set forth by the Insti- 
tute of Education Sciences — the very people respon- 
sible for releasing the study. The Institute of 
Education Sciences, which is part of the Department 
of Education, established the What Works Clearing- 
house (WWC) in 2002"'' to provide researchers, pol- 
icymakers, and the general public with standards by 
which to evaluate research on educational interven- 
tions. The WWC evidence standards require that 
research studies include either random assignment 
or a pretest measure for the research results to be 
considered effective at establishing causality. 

The NAEP assessments do not meet either of these 
criteria. The students who take the NAEP tests each 
year are not randomly assigned into public or private 
schools. In addition, the NAEP does not sample the 
same students or schools during each successive 
administration. This means that the NAEP data do 
not include any pretest or baseline measures that can 
be used to adjust for differences between the public 
and private school populations. 

The best method to draw causal inferences from 
research is to collect data as part of a random study. 
In this type of study, research participants are ran- 
domly assigned to either an experimental group or 
a control group. The experimental group receives 
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some treatment — their environment is changed or 
manipulated in some manner. The control group, 
on the other hand, is not exposed to anything new 
or different and serves as a source of comparison 
for the experimental group. 

Random assignment helps to ensure that the 
experimental and control groups contain similar 
mixes of participants and are generally representa- 
tive of the larger population to which they belong. 
The variable of interest is measured at regular inter- 
vals throughout the duration of the study (includ- 
ing at the beginning and end) to see whether there 
is any difference in outcomes between the experi- 
mental and control groups. In this case, causal 
inferences can be drawn because any difference in 
outcomes between the experimental and control 
groups can be attributed to the treatment experi- 
enced by the experimental group. 

Education studies that include measurement 
over time are much more useful for drawing con- 
clusions about school quality. Education research- 
ers have repeatedly pointed out that a student’s low 
test score at a single specific time may indicate only 
that she is not a good student or that some external 
circumstance influenced her bad performance on 
the test that day. However, if that student’s test 
scores rise over time, it indicates that she is being 
well-served and well-educated by the school that 
she is attending. 

The NAEP data do not sample the same students 
or schools each year. Thus, researchers cannot 
study how achievement has changed over time for 



either individual students or a cohort of students 
attending the same school, whether public or pri- 
vate. As a result, the NAEP data are not well-suited 
to establishing whether a specific math or reading 
achievement outcome is associated with attending 
either a private or public school. 

Evidence on Voucher Programs 

Voucher programs specifically target the aca- 
demic needs of low-income (frequently minority) 
students, who often live and go to school in high- 
poverty areas. These children are frequently stuck 
in persistently low-performing public schools that 
are not meeting their educational needs. 

Currently, six states and the District of Columbia 
offer government-sponsored voucher programs to 
their students. In addition, several other states offer 
privately funded voucher programs.^ These pro- 
grams allocate government or private funds 
directly to parents who then use that money to pay 
tuition costs at a private school of their choosing. 
Voucher programs are a powerful school choice 
tool because they give low-income families the 
opportunity to send their children to private, 
tuition-based schools that they could not afford 
without financial assistance.^ 

Several random assignment studies have been 
conducted on the effects of private school voucher 
programs on math and reading achievement. These 
studies include evaluations of public and privately 
funded voucher programs in Wisconsin,' North 
Carolina,® Ohio,^ New York,^*^ and the District of 
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Columbia, The studies compared the achievement 
outcomes of students who were randomly awarded 
vouchers to attend a private school to the outcomes 
of those students who did not receive vouchers and 
remained in their regular public schools. They all 
reached the same conclusion: Students who received 
vouchers experienced greater math and/or reading 
achievement gains than did the students who 
remained in the public school system. 

Since these studies included random design and 
measured differences in achievement scores over 
time/^ it can be said with confidence that attend- 
ing a private school resulted in significantly greater 
math and/or reading achievement gains for the stu- 
dents who were awarded vouchers. 

Conclusion 

The results of a recent NCES study that finds that 
public school students are performing better then 
private school students should be interpreted cau- 
tiously. The NAEP data used in the study provide a 
snapshot of student achievement at the time the 
NAEP tests were administered. As a result, the NAEP 
data can be used only to describe the characteristics 
of the student population in the United States at the 
time when students took the NAEP tests. 



The NAEP data are certainly not suitable for 
establishing whether a specific math or reading 
achievement outcome is associated with attending 
either a private or public school. Despite this fact, 
the results of the NCES study are being interpreted 
inappropriately to imply that voucher programs, 
which include private schools, are a bad idea. 

The research literature that addresses the effec- 
tiveness of school voucher programs in raising 
math and reading achievement, based on more 
sophisticated methodology, is much more convinc- 
ing and conclusive. Students who attend a private 
school through a voucher program experience 
greater gains in math and reading than do their 
public school counterparts. 

Voucher programs are a powerful school choice 
tool that results in many positive outcomes for the 
students who participate in them. More families 
should have the opportunity and flexibility to 
choose the most effective school for their children 
to achieve success. 

— Shanea Watkins is Policy Analyst in Empirical 
Studies in the Center for Data Analysis at The Heritage 
Foundation. 
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